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SCHEDULE  RISK  DATA  DECISION  METHODOLOGY  (SRDDM) 

1.  INTRODUCTION 

One  of  the  top  priorities  of  the  U.S.  Anny  is  to  make  decisions  regarding  acquisition  programs 
that  will  best  serve  the  Warfighter.  Providing  an  accurate  and  precise  schedule  risk  assessment 
for  a  set  of  alternatives  is  a  key  input  to  the  decision  making  process.  The  Weapon  System 
Acquisition  Refonn  Act  of  2009  requires  trade-offs  among  technical,  schedule,  &  cost  risks 
along  with  perfonnance  to  support  the  Analysis  of  Alternative  (AoA).  This  requirement  remains 
critical  for  all  defense  acquisition  programs  [1]. 

AMSAA  conducts  independent  schedule  risk  assessments  to  support  AoAs  and  other  major 
Anny  acquisition  studies.  A  probability  is  assessed  for  completing  a  given  phase  (e.g.  Milestone 
B  to  C)  within  the  schedule  developed  by  the  Program  Manager  (PM).  The  probabilities  are 
based  upon  historical  data  for  analogous  acquisition  programs. 

Analogous  acquisition  programs  are  historical  programs  or  elements  of  historical  programs 
exhibiting  characteristics  that  are  relatively  similar  to  a  specific  AoA  alternative.  Some  of  these 
characteristics  include  program  type,  acquisition  strategy,  system  capabilities,  critical 
technologies,  and  schedule  drivers. 

AMSAA  developed  a  Schedule  Risk  Data  Decision  Methodology  (SRDDM)  that  determines  if 
enough  historical  data  exists  to  utilize  quantitative  techniques  to  conduct  the  schedule  risk 
assessment.  This  methodology  lays  the  mathematical  and  decision-making  foundation  for  all 
future  schedule  risk  assessments.  Within  SRDDM  are  Monte  Carlo  simulations  and 
mathematical  models  that  build  a  confidence  interval  (Cl)  around  the  probability  of  meeting  the 
PM’s  schedule.  If  the  Cl  width  is  within  the  user  established  tolerance  then  enough  analogous 
programs  exists  to  build  risk  distributions.  If  these  distributions  represent  the  risks  for  the 
alternative  then  the  distribution  associated  with  the  smallest  Cl  width  is  chosen. 

The  main  goal  or  purpose  for  schedule  risk  assessments  is  to  accurately  and  precisely  assess  the 
probability  of  meeting  the  PM’s  schedule,  where  a  low  probability  is  an  unfavorable  outcome  or 
high  risk.  The  importance  or  meaning  of  meeting  this  schedule  (i.e.  mild  to  severe 
consequences)  will  be  detennined  by  Anny  decision  makers. 

AMSAA  has  applied  SRDDM  to  the  Indirect  Fire  Protection  Capability  (IFPC)  and  the  Annored 
Multi-Purpose  Vehicle  (AMPV)  AoAs.  Risk  mitigation  and  trade  space  analysis  can  be 
perfonned  using  SRDDM.  Future  work  includes  developing  event-driven  models  and 
incorporating  technical  risk  assessment  outcomes  and  subject  matter  expert  (SME)  input. 
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2. 


SRDDM  METHODOLOGY 


2.1  Schedule  Risk  Data. 

Schedule  risk  assessments  can  utilize  two  levels  of  historical  data: 

•  Phase-level  data:  High  level  (i.e.  Milestone  phase  dates)  series 
relationships  between  schedule  events. 

•  Event-level  data:  Detailed  (series  and  or  parallel)  relationships  between 
schedule  events  -  this  includes  Work  Breakdown  Structure  and  Critical 
Path  analyses. 

Currently  SRDDM  focuses  on  phase-level  approaches  because  the  data  is  more  readily 
available  than  event  level  information. 

SRDDM  uses  two  data  modeling  techniques  with  the  phase-level  data  resolution  to  create 
distributions  -  Ratio  and  Unadjusted.  The  Ratio  technique  calculates  the  percent  change 
between  estimated  and  actual  dates  from  historical  analogous  programs  and  applies  it  to 
the  PM’s  estimate  for  the  alternative.  The  Unadjusted  technique  uses  actual  dates  from 
the  historical  programs. 

The  figure  below  illustrates  phase-level  Ratio  and  Unadjusted  techniques: 


Analogous 

Program 

PM  Schedule 
(months) 

Actual 

Time 

(months) 

%Change 
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85 

92% 

2 

110 

112 

102% 

3 

23^ 

50 

221% 

4 
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PM's  Proposed 
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Figure  1.  Schedule  Risk  Data 
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2.2  Schedule  Risk  Distribution. 


This  document  focuses  on  phase-driven  schedule  risk  distributions.  Distributions  are 
created  for  each  of  the  alternatives.  These  distributions  can  be  fitted  parametrically  (e.g. 
lognormal)  using  the  “best  fitting”  distribution  or  non-parametrically  (empirical  data). 

There  are  two  data  modeling  techniques  and  the  option  of  “best  fitting”  distribution  or 
empirical  distribution  to  measure  risk.  Hence,  there  are  four  phase-driven  distributions 
as  shown  in  the  tree  figure  below. 


Rat'0  Unadjusted 


Best  Fitting  Empirical  Best  Fitting  Empirical 

Distribution  Distribution  Distribution  Distribution 

Figure  2.  Phase  Distribution  Tree. 

Two  classical  methods  are  used  to  determine  if  the  sample  times  from  analogous 
programs  fit  a  parametric  distribution.  First,  the  Kolmogorov-Smirnov  (KS)  Goodness  of 
Fit  Test  is  evaluated  because  the  samples  of  analogous  programs  are  small  [2],  If  the  KS 
Test  results  for  a  given  distribution  reveals  a  significant  result  [i.e.  P-value  must  be  less 
than  some  significance  level  ( as )]  then  this  distribution  is  considered  as  a  potential 

candidate  for  the  “best  fitting”  distribution.  For  this  hypothesis  test  and  application,  a  = 
10%  is  commonly  used.  Second,  the  Quantile-Quantile  (Q-Q)  plot  is  examined.  A  Q-Q 
plot  graphs  the  quantiles  of  the  empirical  data  against  the  parametric  distribution. 
Quintiles  are  data  points  from  the  cumulative  distribution  taken  at  regular  probability 
intervals.  For  example,  quintiles  with  100  points  are  called  percentiles  and  quintiles  with 
10  points  are  called  deciles.  If  the  data  fits  the  parametric  distribution,  then  the  Q-Q  plot 
should  be  a  straight  line  [2].  If  two  or  more  distributions  have  a  straight  line  Q-Q  plot 
and  P-value  <  as ,  then  choose  the  distribution  with  the  smallest  P-value. 

2.3  Sufficiency  of  Historical  Data. 

2.3.1  Objective  and  Background. 

Evaluation  must  be  completed  to  determine  if  there  are  enough  analogous  programs  (for  a 
given  alternative)  to  apply  the  distributions.  Error  must  be  strongly  considered  regarding 

the  assessed  probability  (let’s  call  it  P  )  of  completing  a  given  phase  (i.e.  Milestone  B  to 
C,  etc.)  within  the  schedule  developed  by  the  PM. 

So,  a  confidence  interval  (Cl)  for  P  is  needed,  where  P  is  the  true  but  unknown 

probability  of  meeting  PM’s  schedule.  P  and  the  sample  are  used  to  build  the  CL 
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Below  is  a  flowchart  figure  of  a  Monte  Carlo  simulation  to  obtain  this  Cl  for  a  given 
alternative  and  for  a  given  distribution: 


I.  Collect  Analogous  Data 


II.  Build  distribution 


I.  Obtain  |p|based 
on  the  Alternative  estimate. 


IV.  Resample  from  the  distribution 
(Draw  a  random  sample) 


V.  Obtain  a 


based  on  this  random  sample. 


Schedule  Proportion 
Sampling  Distribution  (SPSD) 


VI.  ReDeat  steDS  IV  and  V  500+  times  and  obtain  500 

estimates 

Pi,  Pi, . P 500 

VII.  Using  the  Percentile  Method 
with  Bias  Correction  (BC  Method) 
obtain  a  Cl  for  P  (80%,  90%,  etc). 


A  Figure  3.  Confidence  Interval  Process. 

For  a  given  alternative,  this  simulation  is  run  for  each  phase-driven  distribution. 


2.3.2  Schedule  Proportion  Sampling  Distribution. 

The  Schedule  Proportion  Sampling  Distribution  (SPSD)  uses  Visual  Basic  and  @Risk  to 
compute  a  sampling  distribution  for  the  probability  of  meeting  the  PM’s  schedule.  This 
algorithm  uses  Monte  Carlo  simulation  [3],  resampling  methods  such  as  parametric  and 
non-parametric  bootstrapping,  KS  Goodness  of  Fit  testing,  Q-Q  plotting,  and  other 
mathematical  tools.  This  method  produces  a  large  number  of  estimates  for  P.  At  least 
500  simulation  runs  (denoted  at  500  +)  are  required  for  stable  results  -  Step  VI  from 
Figure  3. 

The  first  step  in  this  algorithm  is  to  use  the  sample  of  analogous  programs  (sample  size  is 
n)  to  determine  if  a  distribution  can  be  fitted  using  the  KS  Goodness  of  fit  test  and  Q-Q 
plotting.  If  none  can  be  fitted,  then  we  use  the  empirical  data  as  it  is. 

A  random  deviate  of  size  n  can  be  drawn  from  the  fitted  distribution  (parametric 
bootstrap)  or  a  nonparametric  bootstrap  of  size  n  can  be  generated  from  the  empirical 
data.  A  nonparametric  bootstrap  sample  of  n  programs  is  a  random  sample  of  one 
program  at  a  time  with  replacement  from  the  original  sample  of  n  programs  [4].  Once  the 

sample  is  drawn  we  compute  Px  using  the  data  from  the  first  drawn  sample.  Then  go 
back  to  the  distribution  or  empirical  data  from  the  first  step  and  draw  another  random 

sample  using  either  the  parametric  or  nonparametric  bootstrap.  Now,  P2  is  computed 
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and  this  procedure  is  repeated  500+  times  to  create  the  sampling  distribution  for  the 
proportion. 

This  general  concept  of  resampling  using  information  from  the  original  sample  to  create  a 
sampling  distribution  is  used  in  several  other  AMSAA  developed  methodologies  [4],  [5], 
[6]. 


2.3.3  Percentile  Cl  with  Bias  Correction. 

The  next  step  is  to  apply  the  Bias  Corrected  (BC)  method  [7]  to  this  distribution  of  500  + 
estimates  of  P.  The  BC  method  is  basically  an  adjustment  (for  a  non-nonnal  sampling 
distribution)  to  the  percentile  points  of  the  Percentile  Method.  This  method  adjusts  these 
percentile  points  when  the  mean  and  median  are  not  equal  -  hence  the  method  tries  to 
nonnalize  the  distribution.  In  other  words,  the  Cl  would  shift  toward  the  mean,  left  or 
right  depending  on  where  the  median  is  located.  This  shift  could  be  wider  when  there  is 
more  skewness  in  the  sampling  distribution.  This  BC  adjustment  could  improve  the 
coverage  properties  for  the  confidence  interval  (Cl)  problem. 


A(as)  a  (l-as) 

Let  P  ,  P  indicate  the  100*  asth  and  100*(1  -  as) th  percentiles  from  the  500 
estimates  of  P.  This  represents  the  percentile  method  for  a  2-sided  100*(l-2«s)  CL  The 
lower  and  upper  bound  using  the  BC  method  is  given  by: 


a  (a,) 

P  ,  where  ax  -  <L(2z0  +  z(a,))  ;  Lower  Bound 

a  (a2) 

P  ,  where  a2  =  ®(2z0  +  z"  ) ;  Upper  Bound 

Here  ®(*)  is  the  standard  normal  cumulative  distribution  function  and  z<1_“s)  is  the  100* 
(1  -  as)  th  percentile  point  of  a  standard  normal  distribution.  For  example  zc 95)  =  1 .645 
and  0(1.645)=  .95. 

The  value  of  BC  is  derived  by  the  proportion  of  replications  that  is  less  than  the  original 
estimate  P  .  Here  is  that  value  [5]: 


z0  =  ® 


-i  ,#{Pi<Ph 


500  + 


2.3.4  Wilson  Score  Confidence  Interval  Method. 

When  the  data  shows  that  the  probability  of  meeting  the  PM’s  schedule  is  either  near  0  or 
near  1  then  building  a  sampling  distribution  for  the  probability  can  be  difficult.  In  these 


5 


cases  using  the  Monte  Carlo  simulation  approach  as  documented  in  the  prior  sections 
may  be  difficult. 

In  1927,  Wilson  developed  a  confidence  interval  method  [8]  for  proportions  to  account 
for  extreme  probabilities.  Extreme  probabilities  are  those  probabilities  that  demonstrate 
an  unusual  high  or  low  probability  of  not  meeting  PM’s  schedule,  such  as  near  0  or  near 
1.  The  formula  below  represents  Wilson’s  definition  of  a  confidence  interval  where 
extreme  proportions  exist. 


p  +  4-  ±  z  + 


2  n 


n 


Z 2 
4n2 


72 

1  +  — 
n 


where: 

Z  =  from  the  standard  normal  distribution  with  area  a/2  in  the  tail, 
n  =  number  of  analogous  programs 

P  =  Probability  estimate  from  the  analogous  data. 

o2=  P  *  (1  -  P) 

The  confidence  level  is  100*(1  -  a)%. 

2.3.5  Coverage  Validation  &  Accuracy. 

In  order  to  accurately  build  this  2-sided  Cl  stochastic  model,  enough  sample  data  is 
needed  to  achieve  the  requested  level  of  confidence  (e.g.  90%).  Coverage  models  are 
used  to  validate  the  model  accuracy. 

First  let’s  define  coverage  and  accuracy.  Coverage  is  defined  to  be  the  percentage  of  Cl’s 
that  contain  the  true  population  parameter  P,  where  each  Cl  is  constructed  with  some 
method  at  the  100*(  1  -  as)th  confidence  level  for  a  given  random  sample  of  n  analogous 

programs.  In  other  words,  we  need  to  run  the  inference  method  (Monte  Carlo  simulation 
with  BC  method)  500+  times  (500+  samples  drawn  from  a  parametric  or  nonparametric 
population)  to  obtain  500+  Cl’s.  These  500+  samples  are  not  to  be  confused  with  the 
500+  iterations  from  the  Monte  Carlo  simulation  with  BC  method.  Inspection  is  made  to 
detennine  how  many  Cl’s  contain  the  true  P. 

The  decision  maker  needs  to  detennine  how  much  absolute  relative  enor  for  confidence 
is  tolerable.  The  assessment  of  this  error  is  very  similar  to  the  elicitation  process  to 
detennine  Cl  width  in  the  next  section.  Most  applications  of  this  nature  historically  have 
required  either  80%  or  90%  confidence  with  a  5%  absolute  relative  enor  when  measuring 
acceptable  coverage. 

Accuracy  is  just  a  convergence  rule  for  explaining  the  relative  error  of  a  1 -sided 
coverage.  The  rule  focuses  on  the  speed  at  which  the  relative  enor  approaches  0.  Second 
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order  accuracy  is  defined  as  the  actual  non-coverage  probability  intended  to  be  as  %  for  a 
1 -sided  (1  —  as)%  Cl,  approaches  the  ideal  of  as%  with  error  proportional  to  1/n  [7]. 

First  order  accuracy  would  approach  the  ideal  of  as  %  with  error  proportional  to  -4=  . 

This  means  that  the  relative  error  of  the  1 -sided  coverage  is  of  the  order  0(l/n)  for 
second  order  accuracy  and  0(  ~^=  )  for  first  order  accuracy.  The  BC  method  is  second- 

\jn 

order  accurate  since  the  method  adjusts  the  percentile  points  based  on  the  non-normal 
sampling  distribution.  The  percentile  method  is  only  first-order  accurate  since  it  does 
not  make  any  adjustments  to  the  percentile  points. 

Lessons  learned  from  a  coverage  validation  study  reveals  the  following  results: 

•  At  least  6  analogous  programs  (n)  are  needed  to  perform  any  of  these  confidence 
interval  methods. 

•  If  the  probability  is  extreme  (near  0  or  1)  then  use  the  Wilson  Score  Interval. 

•  If  the  probability  is  not  extreme  then  use  one  of  the  two  Monte  Carlo  methods: 

o  Use  Percentile  Method  if  n  is  10  or  less, 
o  Use  Bias  Correction  Method  if  n  >  10. 

Lessons  learned  demonstrated  that  both  empirical  and  best  fitting  distribution  techniques 
yielded  similar  coverages.  Hence,  choose  the  smallest  Cl  width  when  selecting  between 
these  two  techniques. 

2.3.6  Precision  Error  Tolerance. 

The  decision  maker  (DM)  must  decide  an  acceptable  and  tolerable  width  of  the  CL  The 
assessment  of  the  “tolerance  of  width  of  the  Cl”  is  a  decision  problem  which  requires 
proper  consideration  of  what  happens  to  the  “big  picture  problem”  if  the  endpoints  of  this 
Cl  (namely  the  UB  and  LB)  are  truly  realized.  In  other  words,  the  DM  may  change  the 
decision  as  a  result  of  the  LB  or  UB  occurring.  If  the  decision  is  changed,  then  the 
sensitivity  of  this  width  is  too  large  and  cannot  be  tolerated.  Hence,  the  width  needs  to  be 
smaller.  In  order  to  reduce  the  width,  more  analogous  data  needs  to  be  collected. 

On  the  other  hand,  if  the  DM  does  not  change  the  decision  as  result  of  this  width  then  the 
width  is  acceptable  or  tolerated,  and  enough  data  was  collected.  Keep  in  mind  that 
different  problems  have  different  sensitivities  to  Cl  width.  Sometimes  a  probability  of 
90%  vs.  70%  of  meeting  schedule  will  not  change  the  overall  alternative  level  decision 
(i.e.  both  are  directionally  pretty  good  with  low  risk).  However,  a  probability  of  99%  vs. 
90%  of  a  bridge  breaking  in  the  next  year  could  be  a  decision  changer. 

For  schedule  risk  assessment  applications,  the  main  concern  that  the  decision  maker  has 
is  on  the  LB  because  that  is  where  the  risk  is  contained.  Therefore,  the  risk  is  greater 
when  a  large  width  exists  between  the  mean  and  the  LB  probabilities  compared  to  the 
UB.  The  DM  needs  to  assess  the  largest  width  (mean  to  LB)  that  he  or  she  can  live  with. 

7 


In  other  words,  when  does  the  length  of  the  width  become  an  issue  or  when  does  it  cause 
the  DM  to  re-consider  his  or  her  decision. 

2.4  The  Best  Phase  Distribution. 

Next,  the  distribution  that  best  represents  the  risk  of  the  alternative  is  selected.  Only  the 
distributions  with  an  acceptable  error  for  the  P  Cl  are  considered. 

Determine  the  trustworthiness  of  the  analogous  programs.  The  answer  to  this  question 
lies  in  the  definition  of  “analogous”  that  was  stated  in  the  Introduction.  The  core  of  the 
definition  was  that  the  alternative  and  the  historical  programs  had  the  following  similar 
characteristics:  program  type,  acquisition  strategy,  system  capabilities,  critical 
technologies,  and  schedule  drivers. 

Determine  the  trustworthiness  of  the  initial  forecast.  The  answer  lies  within  two  areas. 
First,  decide  if  the  quality  (i.e.  same  general  forecasting  approaches  are  used)  of  the 
forecast  is  consistent  over  all  historical  programs  and  the  alternative.  Second,  find  out  if 
the  initial  forecast  for  the  historical  programs  were  truly  initial  -  meaning  that  the 
forecast  was  recorded  as  initial  when  in  fact  it  was  really  after  the  program  started. 

If  both  are  trusted,  then  choose  the  distribution  associated  with  the  smallest  Cl  length  for 
P. 
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2.5  SRDDM  Process  Flowchart. 


The  flowchart  in  the  figure  below  is  a  high-level  overview  of  SRDDM. 


SRDDM  lays  the  Mathematical  <&  Decision-Making  Foundation  for  all  future  Schedule  Risk  Assessments 


Collect 
Analogous 
Phase  Data 
fora  given 
alternative 
(Actuals  & 
Predictions) 


Improve  quality  of  SRDDM 
by  developing  event 
driven  models; 
incorporating  technical 
risk  assessment  outcomes 
and  SME  input;  etc. 


Time  to  complete  distribution 


Figure  4.  SRDDM  Process  Flowchart. 


The  steps  to  this  SRDDM  process  are: 


•  Using  the  analogous  data,  compute  the  probability  of  meeting  the  PM’s  schedule  (  P  ). 
For  empirical  data,  this  is  the  percentage  of  the  analogous  data  that  falls  below  the  PM’s 
schedule.  For  best  fitting  distribution,  this  is  the  area  below  the  PM’s  schedule  using  the 
best  fitting  distribution. 

•  Detennine  if  enough  data  exists  to  use  P  as  our  estimated  probability.  A  Cl  should  be 
built  for  meeting  the  PM’s  schedule  using  the  analogous  data. 

•  To  build  this  Cl  we  utilize  one  of  three  Cl  methods  depending  on  the  number  of 
programs  and  the  suspected  probability  of  meeting  the  PM’s  schedule.  These  are  called: 

•  Monte  Carlo  Simulation  Percentile  Method 

•  Monte  Carlo  Simulation  Bias  Corrected  Method 

•  Wilson  Score  Interval 

Details  of  these  methods  were  discussed  in  earlier  sections. 
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•  After  using  one  of  these  three  methods  to  build  a  Cl,  errors  (Cl  width)  should  be 
examined.  The  lower  confidence  bound  (LCB)  is  of  most  concern  because  the  LCB 
represents  a  higher  risk. 

•  If  the  data  and  the  initial  forecast  are  trustworthy,  then  choose  the  distribution  associated 
with  the  smallest  Cl  width. 

•  Risk  mitigation  and  trade  space  analysis  are  conducted  -  discussed  in  Section  5. 
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3. 


NOTIONAL  APPLICATION  #1 


This  is  a  notional  example  to  illustrate  the  SRDDM  process.  There  are  1 1  notional 
analogous  programs  with  this  particular  alternative  within  the  AoA. 

Table  1.  Notional  Analogous  Program  Data  -  Application  #1. 


ACTUAL  or  Unadjusted 

PREDICTED 

Analogous 

Program 

MS  B  -  IOC 

(Work  Days) 

Years 

MS  B-  IOC 

(Work  Days) 

Years 

Actual/Predicted 

Adjusted 

(Work 

Days) 

1 

1787 

6.8 

2586 

9.9 

0.69 

1866 

2 

1995 

7.6 

1763 

6.7 

1.13 

3055 

3 

1806 

6.9 

1348 

5.1 

1.34 

3617 

4 

2437 

9.3 

2177 

8.3 

1.12 

3022 

5 

3050 

11.6 

3785 

14.4 

0.81 

2176 

6 

1784 

6.8 

1813 

6.9 

0.98 

2657 

7 

3218 

12.3 

3262 

12.5 

0.99 

2664 

8 

1816 

6.9 

2002 

7.6 

0.91 

2449 

9 

2158 

8.2 

2110 

8.1 

1.02 

2761 

10 

4435 

16.9 

5458 

20.8 

0.81 

2194 

11 

1755 

6.7 

2044 

7.8 

0.86 

2318 

PM  estimate  =  2700 

10.3 

The  following  results  are  the  outcome  when  SRDDM  is  applied  to  this  notional  data.  The 
definition  and  interpretation  of  these  results  were  explained  in  detail  in  Section  2. 

Table  2.  Application  #1  Results. 


Distribution 

Fit  Results 

P 

Bias  Corrected 

80%  Cl  for  P 

Cl  Length 

Best 

Unadj  /  Fit 

None 

None 

None 

None 

Unadj  /  No  Fit 

N/A 

0.72 

(.45,. 82) 

0.37 

Ratio  /  Fit 

Gamma 

0.63 

(.46,. 8) 

0.34 

Best 

Ratio  /  No  Fit 

N/A 

0.63 

(.37,.  81) 

0.44 

There  was  only  one  distribution  within  Unadjusted  since  the  Goodness  of  Fit  test  did  not 
reveal  a  good  fit.  Note  that  the  Cl’s  for  Unadjusted  /  No  Fit  and  Ratio  /  Fit  are  similar 
only  by  coincidence.  The  Bias  Correction  process  worked  in  opposite  directions  to  cause 
the  Ratio  /  Fit  Cl  to  slide  to  the  right  and  the  Unadjusted.  /  No  Fit  Cl  to  slide  to  the  left. 
This  is  due  to  the  median  value  being  to  the  left  or  right  of  the  mean  value,  as  discussed 
in  Section  2.3.3. 
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For  this  notional  decision  problem,  all  Cl’s  bounds  are  within  acceptable  tolerance  and 
the  analogous  programs  and  the  quality  and  consistency  of  the  forecast  are  trustworthy. 
Therefore,  choose  the  Ratio  /  Fit  distribution  to  compute  the  probability  (.63)  of  meeting 
the  PM’s  schedule  in  10.3  years.  Army  Senior  leadership  advises  that  the  uncertainty  in 
our  risk  assessments  be  recognized,  quantified,  and  considered.  This  is  exactly  what  is 
shown  in  table  2. 

Note  that  the  probability  estimates  for  Ratio  and  Unadjusted  are  different  due  to  the  fact 
that  the  Ratio  data  includes  additional  infonnation  (i.e.  the  initial  prediction  or  forecast). 
This  could  cause  the  Ratio  probability  to  be  higher  or  lower  than  the  Unadjusted 
probability.  Everything  depends  on  where  the  PM’s  estimate  lies.  One  must  not  be 
chosen  over  another  just  because  it  yields  a  more  attractive  result.  If  both  distributions 
are  trusted,  then  choose  the  distribution  associated  with  the  smallest  Cl  width  for  P.  The 
only  reason  to  override  this  rule  is  if  one  of  two  things  occurs: 

1 .  Trust  is  low  in  either  the  analogous  data  or  the  forecast. 

2.  Trust  is  much  higher  in  either  the  analogous  data  or  the  forecast. 
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4. 


NOTIONAL  APPLICATION  #2 


Here  are  results  from  another  notional  example  which  only  has  actual  data  and  three 
alternatives. 


Table  3.  Application  #2  Results. 


PM  Schedule  (in  months) 

MS  B  to  MS  C 

MS  Cto  FUE 

MS  B  to  FUE 

^  . . , 

J 

j  ) 

40 

30 

90  / 

Alternative  A 

#  of  analogous  programs 

10 

10 

8  Z'' 

Probability  of  Meeting  PM  Schedule 

0.90 

0.31 

0.60 

Confidence  Interval  (Cl) 

(0.80,  0.96) 

(0.30,0.62) 

(0.43,  0.70)''' 

Cl  Acceptable  ? 

Yes 

Yes 

Yes 

Alternative  B 

#  of  analogous  programs 

13 

11 

9 

Probability  of  Meeting  PM  Schedule 

0.77 

0.20 

0.45 

Confidence  Interval  (Cl) 

(0.68,  0.87) 

(0.08,  0.29) 

(0.24,  0.58) 

Cl  Acceptable  ? 

Yes 

Yes 

Yes 

Alternative  C 

#  of  analogous  programs 

5 

5 

3 

Probability  of  Meeting  PM  Schedule 

0.60 

0 

0.70 

Confidence  Interval  (Cl) 

(0.30,  0.80) 

(0,0.25) 

(0.30,  0.90) 

Cl  Acceptable  ? 

No 

No 

No 

Not  enough  data  to  make  a  determination 

Notice  that  the  width  of  the  Cl  for  the  3  or  5  analogous  programs  used  in  alternative  C  is 
unacceptable  because  the  Cl  width  is  too  wide  and  yields  too  much  uncertainty  between 
the  probability  point  estimate  and  the  LCB.  Hence,  more  data  is  needed.  One  could 
always  label  Alternative  C  (MS  C  to  FUE)  as  a  high  risk  (probability  =  0)  with  5 
analogous  programs.  Directionally,  this  can  only  improve  with  more  data.  The  Wilson 
Score  Interval  was  used  to  compute  Cl’s  for  alternative  C  since  a  sampling  distribution 
could  not  be  generated  with  these  few  analogous  programs.  Furthermore,  with  a 
probability  near  0,  the  Wilson  method  would  be  used  anyway. 

Recall,  coverage  lessons  learned  shows  that  SRDDM  or  Wilson  method  will  produce 
poor  coverage  validation  results  with  less  than  6  analogous  programs.  Note  that  the  MS 
B  to  FUE  probability  is  not  always  between  the  MS  B  to  MS  C  and  MS  C  to  FUE 
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probabilities.  Even  though  the  sum  of  the  two  proposed  schedules  for  the  sub-phases  is 
90  months,  the  probability  of  meeting  the  schedules  of  these  two  sub-phases  are  sensitive 
to  combinations  for  obtaining  a  sum  of  90  months. 
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5. 


DATA  ALLOCATION  ISSUES. 


Suppose  historical  analogous  programs  from  MS  B-to-MS  C  were  collected  and  this  data 
really  represented  MS  A-to-MS  C.  For  example,  the  programs  may  have  prematurely 
entered  the  acquisition  process  at  MS  B  when  the  technology  readiness  levels  were 
actually  lower  than  claimed.  This  could  result  in  MS  A  to  MS  B  activities  being 
perfonned  during  MS  B-to-MS  C.  An  algorithm  was  designed  to  allocate  some  of  the 
time  collected  in  MS  B-to-MS  C  back  to  MS  A-to-MS  B.  To  do  this,  historical 
analogous  programs  are  collected  that  have  both  phases  and  weighted  average  factor  is 
computed  to  be  applied  to  the  time  in  MS  B-to-MS  C.  This  will  shift  some  time  back  to 
MS  A-to-MS  B. 

This  weighted  average  factor  is  based  on  the  history  of  analogous  programs  with  times  in 
both  phases  and  is  only  an  estimate.  Every  estimate  based  on  data  has  a  Cl  associated 
with  it.  So,  a  Cl  on  the  factor  estimate  is  computed  and  then  all  models  are  reallocated 
and  re-ran  using  the  mean  estimate  and  the  lower  and  upper  bounds  from  the  CL 

Confidence  Intervals  for  Ratio  Means  (CIM4RM)  is  used  to  compute  the  Cl  because  this 
metric  is  a  ratio  mean  [4]. 
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6. 


RISK  MITIGATION  AND  TRADE  SPACE. 


Schedule  risk  mitigation  strategies  are  used  to  reduce  schedule  risk,  i.e.  make  the  risk  less 
severe.  For  example,  the  DM  might  want  to  know  what  can  be  done  to  reduce  the 
schedule  risk  from  high  to  medium  or  low  risk  for  a  particular  alternative  within  the  AoA. 

This  means  that  the  probability  of  meeting  the  PM’s  schedule  (  P  )  would  increase. 

There  are  two  fundamental  ways  to  solve  this  risk  mitigation  problem.  Which  are: 

•  Add  time  to  the  schedule  or 

•  Remove  or  reduce  schedule  drivers  -  this  would  cause  the  “time  to  complete” 
distribution  (based  on  analogous  programs)  to  shift  to  the  left. 


The  following  figure  illustrates  these  two  risk  mitigation  approaches. 


LLsoo 


PM's  estimate 


i _  _  _  _ 


|  Risk  Mitigation: 
Add  additional  ] 
I  |  time. 


Time 


Time  to  complete  distribution 


or 


PM's  estimate 


Risk  Mitigation: 
Remove 
Schedule 
Drivers. 


Time  to  complete  distribution 


Figure  5.  Risk  Mitigation  Strategies. 


Therefore,  visualize  that  P  would  increase  in  both  of  these  strategies  as  seen  in  Figure  5. 
One  can  perform  sensitivity  analyses  with  both  strategies  to  see  how  much  risk  is  reduced 
when  time  is  added  to  the  schedule  or  schedule  drivers  are  removed  or  reduced. 

However,  to  successfully  achieve  risk  mitigation  within  an  alternative,  one  needs  to  know 
how  much  time  (if  any)  should  be  added  to  the  schedule  and  how  much  time  (if  any) 
should  the  distribution  shift.  In  order  to  achieve  this  goal,  one  needs  to  know  the  details 
of  the  schedule  drivers  for  the  analogous  programs  and  for  the  alternative.  These 
schedule  drivers  should  be  similar  for  both  analogous  programs  and  the  alternative,  by 
definition  of  analogous.  Detailed  visibility  to  both  the  analogous  programs  and  the 
alternative  schedules  allows  one  to  see  the  event  details  which  enable  the  possibilities  for 
increasing  schedule  time  or  reducing  events.  For  example,  two  parallel  events  could  be 
unraveled  into  two  serial  events,  hence  adding  time  to  the  schedule.  Risk  mitigation 
strategies  are  usually  perfonned  within  the  alternative. 
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The  DM  is  typically  interested  in  knowing  the  trade  space  between  cost, 
schedule,  and  performance.  For  example,  the  DM  may  want  to  reduce  cost  and 
see  what  the  impact  is  on  the  technical  and  schedule  risk,  and  perfonnance. 

In  order  to  obtain  the  impact  on  the  schedule  risk  assessment  (if  cost  is 
reduced),  one  needs  to  know  what  schedule  events  and  times  are  affected. 

The  trade-off  would  have  to  be  clearly  defined,  and  the  impact  to  the  schedule 
would  need  to  be  provided  by  the  PM,  who  developed  and  knows  the  details  of 
the  schedule  (e.g.  events,  critical  path,  work  breakdown  structure). 
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7. 


CONCLUSIONS  AND  PATH  FORWARD. 


SRDDM  has  been  applied  to  ongoing  AoAs,  along  with  sensitivities  for  risk  mitigation. 
Some  of  these  include: 

•  Indirect  Fire  Protection  Capability  (IFPC) 

•  Annored  Multi-Purpose  Vehicle  (AMPV). 

The  AMSAA  Risk  Team  is  capable  of  using  SRDDM  to  perform  risk  mitigation  and  trade 
space  analysis.  However,  at  the  current  time  AMSAA  does  not  have  full  visibility  and 
knowledge  of  the  schedule  details  and  events.  To  overcome  this  problem,  the  PM  could 
“team  up”  with  AMSAA  to  perform  risk  mitigation  and  trade  space  analysis  since  they  have 
full  visibility,  knowledge,  and  control  of  the  schedule. 

The  AMSAA  Risk  Team  will  continue  to  improve  the  quality  of  SRDDM  in  the  future  by 
developing  event-driven  models  (e.g.  Work  Breakdown  Structure,  Critical  Path,  Correlation 
of  events),  incorporating  technical  risk  assessment  outcomes  (time  distributions  for 
technologies  -  Technology  Readiness  Level,  Manufacturing  Readiness  Level,  Integration 
Readiness  Level)  and  SME  input.  To  execute  and  develop  these  event-driven  models, 
AMSAA  will  work  with  PMs,  SMEs,  contractors  and  any  other  parties  that  can  add  insight 
into  the  event-driven  process. 

The  event  level  approaches  depend  on  two  core  items: 

•  Data  availability  and  Visibility  to  the  detailed  schedule. 

•  Credible  SME’s  to  evaluate  detailed  event  times  and  assess  correlation,  etc. 

The  phase  and  event  level  approaches  will  be  applied  to  an  alternative  to  determine  if  there 
is  a  difference  in  results. 
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